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ABS TRACT 


An extensive and scientifically controlled experiment was designed 
and conducted over a three month period. The analysis of the results 
are accomplished with modern, advanced statistical procedures. The 
effort leads to the conclusion that the Personalized System of Instruc- 
tion (PSI) is demonstrably superior to the conventional lecture/recita- 
tion (CLR) technique for teaching graduate level students (at NPS ina 
certain class of subjects). This paper describes in detail the nature 
of the experiment, the analysis and the benefits to be derived through 
utilization of PSI. The findings of this experiment are directly appli- 
cable to the costly and imperative educational and training missions 


conducted by the Department of Defense (DOD). 
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FOREWORD 


The educational effort supported by the Department of Defense (DOD) 
is very great. Every year billions of dollars are expended to produce 
graduates from hundreds of schools who are hopefully trained and ready 
to perform one of many specialty functions. It is of great interest, to 
those involved in this process, when the chance to improve the quality 
and efficiency of the educational process presents itself. 

An innovative instructional methodology, called the Personalized 
System of Instruction (PSI), burst upon ane academic scene in the mid- 
60's. It promised to markedly enhance the benefits which graduates of 
courses could derive from their educational experience. 

This paper discusses an analytic investigation of the effects and 
benefits which can accrue to students learning via PSI. It is sincerely 
hoped that the findings which it details will assist in the evaluation of 


this new technique for its applicability to the Department of Defense. 





I. INTRODUCTION 


A. LEAD-IN 

Ever since man began teaching others he has wondered about and 
sought methods for improving the efficiency and effectiveness of the 
educational process. In spite of these efforts education has sustained 
little systematic change: teaching has remained essentially an art. 
Controversies have arisen concerning teaching methodology and the 
proportion of responsibility that rests with the instructor and with his 
students. These moot issues have remained a matter for subjective 
analysis over the years because the field of education is only beginning 
to emerge as a scientific discipline. Recently discovered knowledge of 
learning principles derived from experimental analysis of human be- 
havior has had an important and dramatic affect on the development of a 
scientifically based technology of teaching [Skinner, 1968]. One effort 
in this regard is described by Keller [1966], and further research on 
college teaching corroborates his hypothesis that the application of 
behavioral principles in the instructional environment does yield signif- 
icant results in higher education [Ferster, 1968; McMichael and Corey, 
1969; Sheppard and MacDernott, 1970; Alba and Pennypacker, 1971, 
1972; Kulik, 1975]. 

The need for continuing study and development of more effective 


and efficient teaching techniques is manifest, not only because of the 
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growing number of students requiring an education, but also because of 
the complexity, diversity and quantity of the material they must learn. 
In addition budgetary considerations are becoming more constrictive 

as the need for education competes, for funds, with other requirements 
of great urgency or high priority. Thus, the problem of maximizing 
teaching effectiveness most efficiently (i.e., at lowest cost) is a critical 
one. 

Therefore, if the new findings concerning the nature of the learning 
process hold the promise of a true advance in education then they should 
be fully explored. The research effort to be described in this thesis was 
conducted to aid in this regard and, particularly, to address the ques- 
tion of whether a technique termed the Personalized System of Instruction 
(PSI) [Keller, Sherman, 1974] affords specific educational advantages to 


the Department of Defense (DOD). 


iB. SITUATION 

The Naval Postgraduate School (NPS) trains military officers in 
various scientific disciplines at the graduate university level. The 
curricula are demanding and the students matriculate with widely vary- 
ing academic backgrounds. Frequently it has been many years since 
they were exposed to college level academic work. The challenge of 
educating these men as quickly and as effectively as possible is a 
matter of great interest in an era of budgetary restrictions and increas- 


ing technological demands placed upon officers during the conduct of 
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their operational duties. The question of applying PSI as an instruc- 
tional technique within the NPS program was raised by the administra- 
tion and a few courses were designed, written and administered via 

this mode. This thesis analyzes the design, conduct and outcome of one 
such experiment (a course in linear algebra). This experiment and its 
findings are of importance for several reasons. First, the research 
effort was conducted from the start for the purpose of addressing the 
question of PSI's relative effectiveness and its potential. The specific 
output measures employed were carefully selected to provide meaningful 
information that is reliable and valid. Second, the experiment was 
meticulously controlled. All relevant, influential variables were held 
‘constant save for the form of instruction which was left free to affect 

the students, independent of contaminating influences. Third, the 
experiment was a success in that the results were obtained without 
confounding, missing data points or contamination. The output provides 
usable information from which decision makers can draw useful con- 
clusions concerning the expected benefits to be derived from the employ- 
ment of PSI as a teaching mode in DOD educational institutions of many 
varieties. 

It is emphasized that previous experiments presented in the lit- 
erature, in spite of their praise of PSI, did not address the issues at 
the graduate level of education with military officers of high calibre as 
subjects. Further, the great care taken in this experiment to insure 


the existence of matched groups lends significant credence to the 
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impressive statistical results observed across the output measures of 
this experiment. It is genuinely hoped that this research effort will 

further and speed the application of PSI by the military in its vast and 
expensive efforts to educate and train its personnel for the conduct of 


their complex tasks. 
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II. BACKGROUND 


mae CONVENTIONAL LECTURE-RECITATION (CLR) 

The academic year at NPS is divided into four (quarter) segments. 
During each quarter a typical student is exposed to several courses 
(usually three to five) with credit hours ranging from twelve to twenty- 
five eee s. In addition, students in Masters or Doctoral level programs 
work on their thesis or dissertations. Ina course taught via the con- 
ventional (CLR) mode the class meets at designated hours during the 
week for lectures or laboratory work. Assignments, tests, grading 
. and course content, within specified limits, are the responsibility of 
the professor. The class moves through the course ata rate deter- 
mined by the professor and students are afforded the opportunity to 
meet with him for additional instruction if they desire it. The number of 
students ina classroom is typically fifteen and sometimes is as high as 
thirty-five. The amount of "outside class-room work" performed by a 
student is a function of many variables (e.g., the student's personal 
goals, his innate academic abilities, his prior exposure to course 
material, and so forth). This instructional technique is typical of that 
being practiced in most colleges and universities throughout the United 


States. 


14 





B. THE PERSONALIZED SYSTEM OF INSTRUCTION (PSI) 

Consider, for the present, that PSlis a particular mode of instruc- 
tion that differs from CLR in format but not in purpose. That is, PSI 
and CLR have as their primary objective the goal of insuring that stu- 
dents learn as much as possible about the subject matter at hand (i.e., 
theory, mechanics, subtleties, construct, etc.) ina reasonable amount 
of time. Given this supposition it is important to clarify the true nature 
of PSI and to illustrate the manner in which it differs from CLR. 

In March 1963 four psychologists, advocates of the REINFORCE- 
MENT THEORY in learning and adherents of many of the Skinnerian 
Principles of Human Behavior [Friedman, et al 1975] conceived a 
method of instruction that departed from traditional methods which they 
believed were outdated [Keller, Sherman, 1974]. Although no single 
feature of their design was original, the combination of the various 
features was to produce perhaps "'one of the most exciting and radical 
(educational innovations) ever introduced at the university level [Ibid]. 
As the method developed, its skeletal form embodied certain funda- 
mental features: the breakdown of the course into small learning 
"units', self-pacing, repeated testing, and mastery learning before 
exposure to sequential unit material. A PSI course might contain 
lectures, demonstrations, discussions, laboratory/workshop hours, 
and homework. However, the lecture and demonstrations were to be 
infrequent and designed primarily to provide motivation for the student 


rather than to act as sources of essential information. These lectures 
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were to be interesting, informative and memorable -- even entertaining. 
They were producible at suitable intervals and available to students who 
had progressed to a point that optimized their ability to appreciate their 
content. Attendance at these enrichment activities was optional and 
their material was not intended for testing. 

As PSI matured other salient features became manifest. The role 
of the professor evolved to that of an educational engineer, a contin- 
gency manager, whose principal function was to facilitate the learning 
process. The presence and function of proctors (or tutors) to assist 
the professor in grading and answering individual student questions 
became paramount. The proctors were to interface directly with the 
students clarifying complex issues from the written materials or the 
text, diagnosing intermediate student performance, and prescribing 
remedial learning activities designed by the professor. Details of these 
various facets of PSI are explained in Part III. 

Beneath all of the technical characteristics of PSI rest the phil- 
osophical concepts of behavioral psychology that motivate the design of 
the new format. The basic strategy inherent in PSI is one that attempts 
to deal with individual differences in learners: the ability of the learner 
to comprehend the nature of his task, the manner and order of presenta- 
tion to optimally tune to the reception frequencies of the learner, the 
amount of time required by the learner to master the given task (subject 
to the limitations imposed by the length of the academic segments of 


the institution). Frequent diagnostic-progress tests provide detailed 
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feedback to the professor, the tutors and the students: these tests 
reinforce the learning experience and they reveal particular points of 
difficulty (which are accompanied by a very specific prescription of 
what to do about them). No penalty is extracted for non-mastery (save 
the time required of the student to study harder and then re-attempt to 
demonstrate a superior degree of proficiency). A final exam could be 
administered when the student was ready and his grade for the course 
would be a judicious blending of his previous work and his performance 
on the final exam. A student who performed poorly on his final might 
even be afforded an opportunity to take it again. 

The rationales upon which these concepts were developed are that 
the learning of complex materials depends upon the prior enmevery of 
simpler materials [Gangne, 1962, 1965; Moore et al 1973] and that all 
students do not learn at the same rate [Bloom, 1968]. It was the feel- 
ing of the early adherents of PSI that CLR was sub-optimal because it 
did not ensure the acquisition of simpler concepts and skills before it 
demanded the absorption of the more complex. Further, by its very 


nature, CLR pushed (or held) all students through the course material 


at the same rate. 


wee Like VIABILITY OF PSI 


Since its birth in 1963 PSI has been tested and adopted at many 


institutions (e.g., Electricity and Magnetism, Michigan State University; 


Physics, Bucknell University; Mathematics, New Mexico State Univer- 


sity; and others) and has been abandoned only infrequently (e. g., Physics, 
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Massachusetts Institute of Technology (MIT) [Kulik, 1975, Friedman et 
al, 1975; Green, 1971]. However, in no two cases was the testing or 
implementation of PSI exactly the same. Each institution, each profes- 
sor, and each course differed enough to give the various experiments a 
character of their own. This fact does not mitigate the desire nor the 
capacity to examine previous efforts to help establish the benefits of 
PSI or reveal its weaknesses. 

It is important to remember only that any PSI course is built 
around certain characteristics which distinguish it from CLR. Keller 
has summarized these aspects as follows: 


1. The self pace feature, which permits a student 
to progress through the course at a speed com- 
mensurate with his ability and other demands 
upon his time. 


2. The mastery requirement for advancement to 
new material, which ensures that students 
demonstrate understanding of present concepts 
prior to the study of more complex ideas. 


3% The use of lectures and demonstrations as 
motivators, rather than sources of critical 
information. 


4. The stress upon the written word and the 
"hands-on-the material" concept which 
ensures that students practice the imple- 
mentation of the principles they are exposed 
to. 


5. The use of tutors, who interface with the 
students in a variety of ways and provide the 
course with personal/social psychological 
characteristics. 
[Koen, 1974] 
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Bearing in mind that only courses that possess all of these basic 
characteristics are considered PSI, it is appropriate to consider some 
statistics which reveal the demonstrated results achieved with PSI. 
The task is not as straightforward as it sounds. How does one show 
that a teaching method works? Stan Ericksen claims that teachers live 
ina "criterion-free' environment [Kulik, 1975]. If allowance is made 
for the obvious hyperbole, it is easy to concur with this sentiment. At 
least it is apparent that, at the present time, there is no ultimate stan- 
dard against which the efficiency and effectiveness of a teaching process 
can be measured. Among the useful indicators are: 1) end-of-course 
performance; 2) retention; 3) transfer; 4) facilitation; and 5) student 
attitudes. The vast majority of the literature uses one or more of these 
measures of effectiveness (MOEs) to report the objective results of PSI 
experiments. 

Some reports include a sixth measure: cost-effectiveness ~ which 
however, cannot be discussed ina vacuum (i.e., without reference to 
at least one of the other measures). In order to argue the case for the 
cost-effectiveness of one alternative vice another, it is necessary to 
measure the two against some other performance criterion, ascertain 
what is feasible in terms of resources, and then ask which costs less 
in order to achieve its demonstrated results. In this paper the cost of 
PSI is not examined directly. However, no cost-effectiveness analysis 


could be properly conducted prior to the discoveries of this research, 


and, without the results discussed here none would be warranted. 
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A discussion of each of the measures of effectiveness referred to 
or employed in this study follows: 

END OF COURSE PERFORMANCE: Kulik [1975] engaged in 
considerable independent research and after examining over 400 papers 
on PSI instructional innovations he located 31 comparisons of final exam 
scores from PSI and CLR courses. In 30 of these cases final exam (FE) 
performance by PSI groups was superior to that of CLR classes. In 25 
of these cases the difference was statistically reliable (p = .10). This 
means that the probability that the observed differences in group per- 
formance were the result of chance alone, given the hypothesis that the 
two groups were equivalent, was: (.10). Thus the hypothesis of equality 
of group means could be rejected at the (. 90) confidence level. These 
results are not from isolated studies but represent findings across a 
broad range of disciplines, course levels and academic institutions 
[Kulik, 1975]. 

The validity of the FE as a MOE can be questioned. The FE 
may measure, to some extent, the amount" that the students have 
learned but it also reflects the amount that students "cram" and their 
ability as "test-takers". The scores from the FE can hardly be con- 
sidered interval data measures of the benefit any student derived from 
his exposure to the course material, and it is well known that no FE 
taxes a student's knowledge on all areas with equal reliability or 
validity. Nonetheless the pervasiveness of the FE in educational 


research is indisputable. 
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In the past, all great conflicts over teaching innovations have 
been fought on the battlefield of the FE. The results have been: 
stalemate. (For example, a few years ago there were over 80 studies 
published which compared CLR with the ''seminar/discussion'' method 
of instruction. In 51% of these the CLR mode was found to be statis- 
tically superior and in the remaining 49% the seminar technique pro- 
duced significantly better results.) Comparison of other teaching 
methods using FE as an MOE have ended in the same standoff [Kulik, 
1975]. Thus, while the FE is not offered as the "last-word" indicant 
of performance, inthis case, comparison of FE scores shows the PSI 
groups have obtained a clear-cut margin of victory over their CLR 
counterparts. 

RETENTION: To enrich the analysis of experimental results 
some investigators have looked beyond the FE. One very careful study 
at C. W. Post College of Long Island University, New York [Corey, 
McMichael, 1974], showed that the PSI group retained more informa- 
tion for longer periods and that retention differences between the two 
groups grew with time (see Figure 1). 

In five other studies of retention, where the interval from FE 
to retention test varied from 3 weeks to 15 months, the PSI groups 
performed statistically better than students enrolled in CLR classes 
[Anderson and Artman, 1972; Austin and Gilbert, 1973; Breland and 


Smith, 1974; Calhoun, 1973; Nazzaro, Todorov, and Nazzari, 1972]. 
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Figure 1. PSI vs CLR Retention Comparison 
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This figure depicts how PSI students tend to perform better on end of 
course final exams and how they demonstrate greater retention of 
learned material over time. 
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Interpretation of these studies shows that PSI groups seem to be learn- 
ing information in a manner that facilitates recall over a period of time. 
Moreover, PSI seems to promote meaningful learning rather than 
simply the storage of test-related facts in short-term memory. 
Evidently FE comparisons alone,underestimate the magnitude of PSI 
effects. 

TRANSFER: This MOE is more difficult to measure than its | 
predecessors. It involves the determination of whether information 
learned via the PSI mode can be applied more readily to related subjects 
and to practical problems than can information gained ina CLR 
environment. Four studies which investigated this phenomenon reveal 
that PSI produces superior transfer effects. Anderson and Artman 
[1972] found that students who took Physics I via PSI averaged about 
one letter grade higher than CLR students (from Physics I) when both 
groups took Physics Il via CLR. Students at Michigan State University 
who took Engineering Statics ina PSI format outperformed their peers 
from CLR in the follow-on course on Civil Engineering Structures which 
was administered via CLR [Lubkin, 1974]. Engineering students at the 
University of Tennessee taking Fluid Mechanics ina PSI mode received 
better grades than their CLR counterparts when both groups later took 
Hydraulics via CLR [Weissberg, 1974]. Very recent studies from the 
University of California, San Diego and from the University of Maryland 
further confirm the existence of this PSI transfer phenomenon 


[McMichael, 1975]. Various theories attempting to explain this 
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phenomenon abound. However, until more is known about how the 
human brain actually functions, these explanations must remain 
speculative. Nevertheless, PSI was developed from a knowledge of 
accepted psychological principles of human learning behavior and evi- 
dence is growing in favor of the view that PSI fosters a type of learning 
that goes beyond that which is characteristically gained from CLR. 

FACILITATION: The question of how well PSI students per- 
form in their other (concurrent) courses is an important one. PSlis 
often charged with requiring an inordinate amount of its students’ time 
and with encouraging them to procrastinate. 

A student survey conducted at Ohio State University's College 
of Medicine addressed this issue [Schimpfhauser, Horrocks, Richardson, 
Alben, Schumm, and Sprecker, 1974]. All of the medical students con- 
sidered were enrolled in the same three courses (biochemistry, anatomy 
and physiology). A group of these students took biochemistry via PSI. 
All other courses were taken conventionally. Not surprising (to the 
researchers), the PSI group did better in biochemistry. Somewhat 
phenomental however, was the result that the PSI students did better 
than the other students in all their courses. This finding is quite signif- 
icant since the PSI group was randomly selected and shown to be statis- 
tically equivalent to the CLR group over such relevant variables as Grade 
Point Average (GPA) and Medical College Aptitude Test (MCAT) scores. 


Most importantly this finding shows that if PSI does make greater demands 
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upon student study time, it does not do so at the expense of performance 
in other courses: onthe contrary, PSI, with its requirement for mastery, 
its frequent diagnostic tests, and its unique student-tutor relationship, 
may foster efficient study habits and imbue in the student a desire to 
achieve higher standards of performance. This desire to do superior 
work seems to carry over into all student (academic) activities. Addi- 
tional data collected by Born and Davis [1974] support that point of 

view. They found that students do spend more time working ona PSI 
course than CLR students but that the PSI students spent this time in 
markedly different ways. Most notable was the absence of time spent 
listening to lectures and trying (simultaneously) to take meaningful 
notes. Such activity is largely self-defeating in that the conduct of one 
action detracts from one's ability to do the other. PSI effectiveness is 
apparently not derived from the amount of time spent studying but rather 
in the manner in which this time is spent. 

ATTITUDES: Since PSI was conceived in the minds of behav- 
ioral psychologists it is not surprising that student perceptions of the 
course, their sense of personal accomplishment, and their preferences 
for further learning via PSI was to be an important consideration. 

Every human experience carries with it some learning about the nature 


of oneself. Certainly this is true with educational experiences. Every 


course that the student takes contributes to his self-awareness and his 


concept of education. This conceptual maturation process is called 
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attitude growth. Some investigators [Silberman, Parker, 1974] report 
that PSI contributes to the development of positive attitudes, but objec- 
tive analysis is difficult and often incomplete. Many researchers have 
collected open-ended evaluations from their students, and have classi- 
fied their comments as "'positive'' or "negative. '' Judging from published 
documents the percentage of students commenting unfavorably on the PSI 
format is very small [approximately 4%, Gallup, 1970; Green, 1970]. 
When students are asked directly whether they prefer PSI to CLR the 
results are typically in favor of PSI. At the College of Engineering, 
University of Texas at Austin, about 70% of the students in a PSI course 
preferred that mode to CLR: 20% considered the two modes equivalent 
[Stice, 1971]. Still other reports compare student responses on end of 
course evaluations. In almost all cases PSI appears highly attractive to 
students. 

The results of these comparisons, across the output measures 
discussed thus far, were presented by Kulik [1965] and are summarized 
mae table |. 

Some critics of PSI have attempted to explain away these 
results by claiming that the HAWTHORNE EFFECT (i. e., an enthusiastic 
response to something novel) is in operation. However, contrary to this 
hypothesis, there seems to be a slight increase in the effectiveness of 
PSI as it matures [Kulik, Kulik, and Miholland, 1974; Roth, 1973], so 


a sort of anti- HAWTHORNE EFFECT may actually be at work. 
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Table 1. Comparisons of PSI and CLR Groups 


Favor PSI Favor CLR TOTAL 

Statistical Non-Sig Statistical Non-Sig 

Sig (p=. 10) Sig (p=. 10) 
FE 25 > 0 1 31 
Retention 6 0 0 0 6 
Transfer 4 1 0 0 5 
Facilitation ] 0 0 0 1 
Attitudes 6 ] ] 0 8 


This table reveals the results of comparisons between PSI and CLR 
groups taking various courses together across the nation at many 
different schools. Note: in thirty-one experiments where final exam 
(FE) scores were used as an output measure that thirty differences 
in mean score were in favor of the PSI students. Of these thirty 
differences, twenty-five were so large as to lead one to reject the 
hypothesis of equality of performance at the 90% confidence level. 
Only one of the experiments produced results favoring CLR, in the 
FE category, and that difference was too small to be considered 
statistically significant at the same confidence level. 
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There are other arguments that charge PSI materials with 
"teaching the test.'' These charges appear weightless when one considers 
the aforementioned results concerning retention, transfer, and facilitation. 
Still others submit that the observed results are true but not of any real 
consequence. Consider an average student, who may take his physics 
course, for example, via CLR or PSI. If he takes a typical lecture 
course, his achievement in Physics will place him at the 50th percentile 
on a standardized test. He is an average student taking an average 
course. If he takes the same course via PSI, he will achieve at the 75th 
percentile in the same standardized test. Additionally, he will retain 
what he learned for a longer period, will find that he can transfer the 
information to related courses with greater ease, will learn a great deal 
about efficient study habits and test taking, thus increasing the likelihood 
that his grades will rise in other courses. This increment - the PSI 
effect - is what the technique has to offer the individual student. If this 
effect for one is multiplied by one-hundred, the number of students a 
typical professor may have ina quarter, then some conception is gen- 
erated regarding what PSI has to offer the instructor, the academic 
institution and the taxpayers who support the educational effort. 

For most students, most professors, and most decision makers 
these effects will not seem small. 

At this point is is appropriate to consider what PSI really does 
to the instructor's position. Obviously an educational innovation as 


radical as PSI would be expected to alter substantially his traditional 
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role. The question is actually whether the new role is for the better. 

Instructors in PSI courses find that they cease being broad- 
casters of information and that they are no longer "spotlight performers. '"' 
Instead they become managers of a system and they facilitate learning in 
others. This point about the systematic character of PSI is noteworthy. 
Certainly a truly superior CLR course can be constructed and conducted 
by an outstanding professor. Such a man could duplicate his perform- 
ance many times (assuming no personal distractions affected his ability 
to concentrate and prepare himself consistently) but because his course 
is not systematized it is doubtful that another man could duplicate his 
efforts with the same course. Not so with PSI. Once the course has 
been written and validated the professor becomes a system orchestrator. 
He guides his tutors and develops these subordinates as teachers. He 
prepares his occasional lectures/demonstrations/workshops and, most 
important, he can devote himself to helping individual students with 
particularly significant difficulties. Further, the professor must watch 
the "dials'' in the process: at what rate are students progressing, which 
students are in trouble and need his special guidance, what logistics 
problems exist and how can they be mitigated. Thus the competent 
professor finds that: 


a. He knows where each student stands at all times through- 
out his course. 


b. He can give individual attention to students who need him. 
c. His students are really mastering his course. 
d. He can spend more time preparing his occasional lectures. 
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e. He can take responsibility for developing his tutors as 
effective teachers. 
f. His administration is pleased that he can teach many 
students effectively, at low cost. 
[Green, 1971]. 
The professor teaching a PSI course for the first time might feel strange 
as an "educational engineer, '' but if he feels his role has diminished 
because he is not lecturing from his pulpit, ona daily basis, he has 


perhaps forgotten the central issue in education: the advancement of the 


student in the discipline of interest. 
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IW, THE. EXPERIMENT AE NES 


A. INTRODUCTION 

Due to the many indicants of the benefits to be derived from PSI and 
its sound theoretical basis [Kingsley, 1971] a few courses were recently 
developed at NPS for presentation both on and off campus. Considerable 
interest ve the merit of these innovations was generated within the 
faculty, administrative offices, and the student body. Individuals from 
each of these groups formed opinions based upon varying degrees of 
exposure to the PSI technique and much subjective analysis was conducted. 
The whole issue is of considerable interest to the DOD in general. DOD 
has indicated that NPS should prepare itself for the eventuality that the 
length of time that students stay on campus for degree completion might 
be cut (by up to 15%). The rub is that the quality of the education stu- 
dents receive should not be diminished. The requirement to reduce the 
length of a student's stay on campus is generated by the desire to cut 
costs. Hundreds of millions of dollars are allocated annually by DOD 
for training and education. This allocation is warranted as a result of 
the real need for technically proficient personnel at all levels of activity, 
including management. However, opportunities to reduce cost levels 
while maintaining or even improving the quality of the educational 


process are actively sought ona perennial basis. 
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Beginning in Quarter II, FY 75 a course written by Professor 
Maurice D. Weir of the Department of Mathematics was offered to 
selected students from several disciplines. The course was presented 
in the PSI mode and adhered to all of the principles of the Keller Plan 
[Keller, Sherman, 1974]. After its second presentation in Quarter III, 
FY 75 there was considerable feeling that it was successful. Students 
were learning well, as evidenced by their scores on their FE as com- 
pared with those of students in CLR classes who took the same exam. 
Additionally, the PSI students expressed appreciation for their exposure 
to the new mode of instruction and seemed to genuinely like the course. 
However, many hard questions could only be answered by subjective 
Opinion. Most of these questions had been addressed by experimenters 
working with other courses at different institutions (as indicated pre- 
viously), but it was considered appropriate to answer these questions 
in the unique academic environment at NPS. The major reasons were 
that the students at NPS were not characteristic of students at most 
civilian institutions, that the program of instruction and the work load 
at NPS were considerably more stringent, and because little of the 
previous research was conducted with graduate students as subjects. 

In order to gainan analytic and objective understanding of the 
effects of PSI at NPS a carefully controlled experiment was designed 
around Professor Weir's course. This course entitled: ''Computational 


Linear/Matrix Algebra (MA 2045)" provided the basis for the experiment 
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and the mode in which it was taught constituted the stimulus. The per- 
formance of the students, as measured across specific output variables, 
constituted the response to, or the effect of the stimulus. 

In this chapter the design, conduct, results and analysis of that 


experiment are discussed. 


feldkt STUDENTS, THE PROFESSOR, AND THE TUTORS 

All of ene subjects in this experiment were students in their first 
quarter at NPS. Their basic disciplines were the technical sciences 
(e.g., electrical engineering; computer science) and they were working 
towards the degree Master of Science. All of them were required to 
take MA 2045 and their selection as members of the PSI group or the 
CLR group was determined by the administrative offices of NPS 
(scheduling criteria were the only factors affecting the placement of 
students in one group or the other). Asa result twelve U.S. students 
and four International students were enrolled inthe PSI group. Sixteen 
U.S. and five International students were scheduled to take the course 
via CLR. The only assumption concerning these groups was that they 
were homogeneous or matched (i.e., that across all relevant character- 
istics the two groups constituted random, independent samples drawn 
from the same population). It was felt that this was a valid assumption 
since all students were military officers with fundamentally similar 


academic and professional experiences. They aspired to similar goals 
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and had demonstrated personal competence within their career fields 
prior to selection for attendance at NPS. Hopefully, individual differ- 
ences between students would balance out within the groups so that the 
group characteristic centroids would be co-located (ina statistical 
sense). By this it is acknowledged that the characteristic vector which 
would completely describe something as complex as a group of students 
would be multi-dimensional. Since each group consists of several 
equally complex individuals, the student characteristic vectors, if 
plotted in the appropriate space, would forma group of points. The 
center (centroid) of each group should be in approximately the same 
position for both groups if in fact they are matched. 

In order to support the validity of this assumption it was decided, 
a priori, not ee incorporate the performance of the International students 
in the results of the experiment. This was done because the level of 
language proficiency varied greatly among these men. To have con- 
sidered their performance without accounting for language fluency would 
have introduced an uncontrolled variable in the experiment which would, 
perhaps, have biased the results. In addition several measures of 


U.S. student attributes were obtained and objectively compared to 


determine the relative validity of the hypothesis of group equality. 


*The author analyzed the performance of the international students 
to satisfy his own curiosity. If he had incorporated their performance in 
his overall analysis the result would have been to even further support 
the conclusion of increased effectiveness of PSI over CLR. 
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c= ANALYSIS OF STUDENT CHARACTERISTICS 
Six measures of student attributes were obtained prior to any 
instruction. They were as follows: 


a) Performance ona pre-instructional linear/matrix 
algebra entrance examination; 


b) two scores attained ona standardized test of mental 
maturity [Sullivan, Clark, Tiegs, 1963]; 


c) . two scores attained from a standardized inventory of 
personality traits [Thorpe, Clark, Tiegs, 1953]; and 


d) anumerical representation of the degree of exposure to 
college level mathematics instruction prior to matricula- 
tion at NPS. 

The pre-instructional inventory examination was written by 
Professor Weir: it tested the ability of the students to perform basic 
manipulations with vectors and matrices. It was administered to deter- 
mine if the ability of individual students, in one group or the other, was 
sufficiently developed to permit them to perform in a superior manner 
in MA 2045 regardless of the form of instruction to which they were 
exposed. The test of mental maturity [Sullivan et al, 1963] was pub- 
lished by CTB/McGraw-Hill. It was designed to provide information 
about the functional human abilities that are basic to learning, problem- 
solving and responding to new situations. The scores produced by this 
test are similar to those traditionally known as intelligence quotients 


(I1.Q.). The personality inventory [Thorpe, et al 1953], also published 


by CTB/McGraw-Hill, was administered to help identify the status of 
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i will 


factors in student personality and social adjustment usually designated 
as "intangibles" (e.g., individual motivation, self-confidence, self- 
awareness, etc.). These factors traditionally defy appraisal or diagno- 
sis by means of ordinary ability and achievement tests. Both tests 
published by CTB/McGraw-Hill had been checked for reliability and 

had been validated over several years. National norms and quantiles 
were available and it was felt that these tests would provide a satisfactory 
measure of the variables they were designed to examine. The numerical 
measure of prior exposure to mathematical subjects was obtained using 
an algorithm that gave points for the number of, as well as the types of, 
mathematics courses a student had taken. Then it discounted these point 
totals by a scalar that was sensitive to the number of years since the 
courses were taken. (The data obtained from these tests are available to 
the reader in Appendix B, Sequence 1). 

The results of this pre-instructional testing were subsequently 
analyzed according to several statistical tests of varying power (depend- 
ing upon the nature of the various assumptions demanded by the different 
tests). First the data was depicted in histogram form. These graphs 
reveal that the data, over the variables taken separately, distributes 
itself in a similar fashion over the groups. However, there is no 
indication that the nature of this distribution is known. Therefore, the 
null hypothesis (Ho): that the groups are independent random samples 


(of size: ™Mysi ea Dar 16) drawn from the same population, was 
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tested first according to the non-parametric Kolomogorov-Smirnov (K-S) 
two-sample criterion, with a computer program that determined the 
exact probability of the occurrence of the observed difference in group 
means (or a larger difference) [Kim, Dennick, 1970; Learmonth, 1975] 
(see Appendix B, Sequence 2a,b). The outcome of this testing is shown 
in Table 2. If, for example, one examines the result on the pre- 
instruction inventory test, the value of the test statistic is reported as 
.208. The probability that an observation this large could occur by 
chance alone, given that A, is true, is .198. This means that Ae 

could be rejected at the 19. 8% confidence level. This level of confidence 
is too low for most statisticians to feel comfortable about rejection 
(normally one does not reject below the 90% level of confidence). Note 
that only with the Prior Exposure Variable does the observed difference 
between group means approach statistical significance (i.e., 88.7%) 

and the data shows that this difference is in favor of the CLR group. 
Thus, it is felt that the assumption that the groups are matched isa 
valid one. 

However, since it is known by statisticians that non-parametric 
tests lack power, ''confidence" in statistical findings occurs if one is 
led to reject Ho not accept it. While the desire here is to establish 
Ho: there are no known tests to do so. Nonetheless, armed with the 
non-parametric results it is worthwhile to subject the same data to the 


relevant parametric tests. If these tests also deny us, with sufficient 
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Table 2 


Variable Test Statistic P(D mn = 2S)e= ee 


Pre-Instruction 
Inventory Exam ~ 20S - 198 


Mental Maturity 
Language 292 A558 


Mental Maturity 
Non-Language oe . 636 


Personality Traits 
Personal .229 Ome 


Personality Traits 
Social woed 7ovo 


Prior Exposure 
Mathematics Courses .417 . 887 


mhese results indicate that H, could not be rejected across any variable 
at normal levels of significance employed by statistical researchers. 
Only across the "prior exposure" variable does the difference between 
groups approach statistical significance ( p =.113) and that difference 
is in favor of the CLR group (see Appendix B, Sequence 1; 2a,b). 
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confidence, the ability to reject H,, then even further evidence is 
presented to support the assumption of matched groups. 

The next test employed was the parametric ''student's t(26) » or 
the univariate F-ratio mL 26) (one-way Analysis of Variance, ANOV). 
This test also examines the difference in the observed means of groups 
across variables (one at atime). It requires two assumptions, in 
addition to that of independence between observations as required by the 
non-parametric, Kolomogorov-Smirnov (K-S) test. These are: a) that 
the underlying distribution of the random variable is normal and b) 
that the variance of the random variable is the same for both groups. 
These are strong assumptions, justified by common usage and the 
design of several of the measures of effectiveness. When they are 
satisfied the test has great power. Nonetheless, the results of this 
analysis are in agreement with the K-S test (see Table 3). This table 
provides two measures of the significance of the observed differences 
in group means across the variables (take one ata time). The first 
is Wilk's Lamda ( X ), which varies between zero and one. As(j} ) 
approaches one, the probability that the group means are disparate 
goes to zero. The other is the value of the test statistic 0 ney: Kor 
this statistic to have any reasonable significance it must be greater 
than one. Again, only across prior exposure is this the case. The 
observed value 2.5355 would enable one to reject ae at some confidence 


level below 90% which is considered too low a level for rejection. 
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Variable 


Inventory 


Mental Maturity 
Language 


Mental Maturity 
Non Language 


Personality 
Personal 


Personality 
Social 


Prior Exposure 


Table 3 


Wilk's Lamda 


- 9668 


- 9986 


- 9710 


- 9936 


~9925 


rele 


2 
ela on = (2G) 


- 8929 
0353 
sO 
. 1669 


- 1957 


co, 309 


Significance 


peo 


These results indicate that the difference between group performance 
across the variables is not significant except over the Prior Exposure 
where H, could be rejected at p 


120: 


The difference between group 
performance on this variable is in favor of the CLR Group. 


(* implies that statistical significance at .10 level is not reached). 
(See Appendix B, Sequence 3). 
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Each of these tests examines the input variables singly. In order to 
look at the combined effects of all variables taken together, a two-way, 
nested, factorial ANOV was performed (Table 4). Again, the results 
support the hypothesis of equality of group means. Table 4 reveals that 
the "between students" effects are not significant and neither are the 
effects between groups. As might be expected there is an effect across 
variables (due to different scales of measure) and there is a cross effect 
indicating that neither group performed consistently better or worse 
across all variables (see Appendix B, Sequence 4). 

Finally, in an effort to determine the existence of a discriminating 
capability across the student input characteristic variables and to take 
advantage of the power associated with a procedure allowing the step- 
wise inclusion of the variables, a technique known as discriminant 
analysis was employed [Klecka; in Nie et al, 1975]. The use of discrim- 
inant analysis in this manner is innovative and valid. All of the data was 
input to the program. The computer calculated the prior probability of 
a student being in one group or the other (based upon group size (m,n)). 
It also read the scores of each student on each variable. Armed with 
these facts it computed a discriminant function (see Appendix B, para- 
graph A4 for a detailed explanation) and attempted to classify students 
into their proper group. Failure to do this well would indicate little 
differences between the groups but the ability to do so better than expected 


(if the only difference between groups was, in fact, their size) would 
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indicate that the variables were discriminating, as desired. The results 
were most rewarding. It was found that significant differences did not 
exist between the groups but that students could be classified with 78.5% 
accuracy. 

This compares favorably with the 57% accuracy expected if the 
variables were not discriminating at all, and the program was classify- 
ing based upon prior knowledge of group size (i.e., 16/28 = .57). A 
slight variation between the groups that might have had an impact was 
the different number of average credit hours in their course loads. The 
CLR group averaged 17.54 hours and the PSI group took a mean weekly 
load of 19.39 hours. This chance occurrence was a result of academic 
scheduling from the administrative offices of NPS. As a result three 
PSI students opted to drop MA 2045 prior to the end of the first week of 
instruction. If the CLR had been burdened with the additional hours it 
would have been necessary to make experimental adjustments. Since 
the PSI group had the disadvantage it was decided to permit the groups 
to continue in the investigation, thus enriching the potential for gathering 
information on the effects of PSI ee the adverse conditions ofa 
relatively heavy academic schedule. 

One other source of possible variation, the effect of the professor, 
was controlled by having the same man assume responsibility for the 
conduct of both classes. He would teach the CLR group in the traditional 


manner and he would manage the PSI group according to the principles 
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of the Keller Plan. The professor (Alan B. Shorb, Ph. D) was chosen 
for the critical role because he had taught MA 2045, via CLR, many 
times and was considered particularly capable (according to opinion 
expressed by his previous students and his faculty peers as well). In 
addition he had attended a three-day workshop in PSI conducted in the 
winter of 1975 at NPS. The workshop had been led by Professor Ben A. 
Green, Jr. of the Institute for PSI, Georgetown University. Further, 
Professor Shorb was familiar with the content of Professor Weir's 
written PSI course materials and had expressed no a priori preference 
for teaching via one mode or the other. He had not taught MA 2045 via 
PSI prior to the experiment and it was felt that this fact would enable one 
to determine whether Professor Weir's course was truly systematized 
(i.e., could a competent professor teach successfully via PSI with 
course materials he had not worked with before). The same text 
[Steinberg, 1974] was used for both groups. 

The tutors (two) were selected from the student body. One was an 
electrical engineering major in his 3rd quarter of an eight quarter 
program. He had previously taken MA 2045 via PSI from Professor 
Weir. The other was the author of this thesis, an Operations Research/ 
Systems Analysis major in the 7th quarter of an eight quarter sequence. 
He had taken MA 2045 in his 2nd quarter via CLR. Both tutors were 
thoroughly briefed on their responsibilities by Professors Weir and 


Shorb. 
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me LHE CONDUCT OF THE CLR CLASS 

The class met for lectures at a pre-designated time for fifty minutes, 
three days a week (Monday, Wednesday and Friday from 1110-1200 hours). 
The professor gave reading assignments from the text, homework prob- 
lems, and offered the students an opportunity to work on two outside 
projects (for extra credit to be applied to the FE score). He announced 
the objectives of the course on the first day of class and explained to 
students how they would be graded. The students were also told that 
they were the control group in an experiment but that this condition would 
in no way affect their instruction or their grade. They were given near 
pre-instruction tests and great care was exercised to motivate them to do 
their best. They were assured, for example, that their scores would not 
be publicized but that they could be counseled on an individual basis, 
if they so desired, as to the significance of their performance on each 
test (92% of the students desired such counseling and showed great 
interest in their scores). 

Each class hour progressed inthe traditional manner. The instructor 
lectured each day, answered questions from the class, and administered 
two ''mid-term'' examinations, each of which consumed one class period 
and contributed approximately one-fourth to the course grade. The FE, 
plus any extra credit, contributed approximately one-half to the course 
grade. These proportions are not exact as the professor reserved the 


right to modify any grade as his subjective evaluation dictated (only the 
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course grade was so affected; the tests and extra credit problems were 
graded as objectively and fairly as possible, with consideration for the 
awarding of partial credit on questions that were not answered perfectly). 


All tests were closed book with no notes available to the students. 


EB. THE CONDUCT OF THE PSI CLASS 

This class was conducted, as closely as possible, according to the 
principles of the Keller Plan. All materials (study guides, diagnostic 
tests, etc.) had been prepared in advance, and all logistical and adminis- 
trative considerations had been accounted for prior to the first day of 
instruction. The students were advised that they were the experimental 
group and would receive their instruction via PSI. They were offered the 
opportunity to withdraw from the group without prejudice if they so 
desired. None withdrew initially. The PSI technique was explained in 
great detail, both verbally and ina written(handout) "Course Policy 
Statement, '' and all questions concerning the conduct of the course were 
answered. The students met their tutors and were told that they could 
employ either of them as they so desired. It should be noted that one 
tutor could have sufficed for a class of this size (m= 12). However, it 
was felt that two tutors would enable the students to pick the man that 
they could most easily relate to, would reduce the requirement for the 
instructor to ''teach'' the PSI students, and would reduce the amount of 
time that a tutor was unavailable for consultation because he was grading 


a diagnostic examination. 
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The PSI students were given their pre-instruction tests and were 
given the same careful instructions and motivational speech as were 
their CLR counterparts. Once the course began the students progressed 
through the material at their own pace. When they had completed the 
work required ona unit in the sequence, they sought out a tutor and 
requested the diagnostic examination for that unit. They could do this 
between 0900-1030 hours on Mondays, Wednesdays and Fridays. All 
work on these exams had to be completed by 1100 hours. During these 
hours the instructor and tutors were present ina designated classroom 
and were available for consultation or test grading. At other times during 
the week tutors could meet with students at mutually acceptable times for 
consultation but not to administer tests. 

After a student completed a diagnostic test he brought it to the tutor 
for immediate grading. The test was gone over thoroughly in the pres- 
ence of the student and if the student received a score of 90% or better 
he was permitted to proceed to the next unit. Otherwise he was advised 
that he had not ''mastered'"' the material and would have to retake a 
similar exam. Exams could be retaken the same day but only after a 
delay of thirty (30) minutes. (Since exams were usually forty-five min- 
utes long and required approximately ten minutes to grade; this reduced 
the likelihood that two exams could be taken in one day. This was done 
in order to motivate students to thoroughly prepare for each exam and to 


master them the first time.) It should be noted that even if a student 
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received a score of 90% or better, any errors he made were discussed 
at length to ensure that he completely understood the material. If he 
failed to demonstrate mastery the first time, the second exam given was 
different than the first but similar in design. All exams were closed 
book and no notes were permitted in the examination room. 

The professor presented three optional lectures during the conduct 
of the course. The information contained in these lectures was designed 
to motivate the students and to enrich their appreciation for the applica- 
bility of the information they were learning to the solution of practical 
problems. Students could attend these lectures regardless of where they 
stood in the course but were advised that their understanding of the lec- 
ture material would be a function of their progress to date. 

As inthe CLR group, the PSI students were advised of the extra- 
credit assignments available to them and that completion of these prob- 
lems would enhance their FE scores. 

The grading of the PSI group was quite different from that of the 
CLR group. At NPS an average grade of B (3.00) is the minimum for 
graduation with a Master's degree. Thus, it behooves each student to 
set his standards at that level, at least. Accordingly, the PSI group was 
told that if they completed all of the required units, mastering (90% or 
better on the diagnostic tests) each in turn prior to the end of the quarter, 
and if they took the final exam (scoring at least 90%), they would receive 


a course grade of A (4.00). Otherwise their course grade would be 
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diminished according to their performance on the final exam (a course 
grade of B, for example, required a score in the 75-79% range). The 
final exam could be taken at any time, so that a student who completed 
the course early would not have to wait to take it during "finals week." 
To give meant where credit is due, students who completed all units 
(recall this implies mastery) would receive at least a B- (2.66) even if 
they did not take the final or performed very poorly on it. This scheme 
was adopted because it was believed that a student who went through the 
entire course (taking thirteen exams along the way, scoring at least 90% 
on each one) deserved at least a B- (2.66). Moreover, a grade of B- 
was a slight penalty (in view of the B (3.00) minimum graduation 
requirement) which would encourage each student to take the final exam 
and perform to the best of his ability. The fact that every student took 
the final exam and demonstrated a high level of performance on it is 
supportive of the reasonableness of this kind of scheme for NPS students. 
It is acknowledged that the self-pacing aspect of PSIlis mitigated by 
the requirement that students complete the course in not more than twelve 
weeks. However, in this experiment practicality prevailed. A "quarter'"' 
is only so long and MA 2045 is a required course in these students' 
curricula. The school administration could not allow a student to con- 
tinue the course beyond the end of the term without significant revisions 


in its curricular programs, 
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Pee tHe OUTPUT MEASURES OF EFFECTIVENESS (MOEs): PSI vs GCblR 
1. Percent of students completing the course 

As the course began in earnest, nine of the original twelve 
students inthe PSI group remained. The three students who dropped the 
course, inits first week, did so for personal reasons relating to the 
large academic loads they were scheduled to take and their own evalua- 
tion of how they could best meet the requirements of NPS for graduation. 
One of the students had been away from higher education for twelve years, 
for example, and feared that devotion of the time required by a PSI course 
would inhibit his ability to get an acceptable grade average in his other 
courses. A second student had taken an advanced course similar to MA 
2045 in undergraduate school (USNA) and felt he could drop MA 2045 in 
his first quarter with the least effect on his ability to satisfactorily pro- 
gress through his other courses. The third student advised his profes- 
sor that he wished to "'audit'' the course, that is, perform the required 
unit exercises at his leisure and avail himself of the tutors, but not 
receive course credit. This would obviate the necessity of taking the 
time consuming examinations and probably ensure him of a high grade 
when he did take the course for credit at a later time. Thus, no Pol 
student dropped the course due to dissatisfaction with the mode of 
instruction. All of the nine students who actually began the course were 
able to complete it well within the time constraints. 

All sixteen CLR students completed the course for credit. Thus, 


across this criterion the two groups performed equally well. 


50 





2. Final Exam Scores (FE) 

The FE was a two-hour closed book comprehensive test that 
covered all the material presented during the quarter. It was written 
by the course professor, in collaboration with the Chairman of the 
Mathematics Department, and reviewed by the tutors and other mathe- 
matics professors. There were two similar versions. The first was 
given one week early to those PSI students that completed the course 
prior to the end of the quarter. The second was given to the remaining 
PSI students and to all of the CLR students at the scheduled final exam 
period designated by the academic offices. Both tests were administered 
and graded by the tutors of the PSI course. The same grading scheme 
was used for both groups. Afterwards the tests were reviewed by the 
professor. He lowered no grades but reserved the right to give additional 
credit where he found the tutors had been too strict. Ironically, this 
occurred on two papers belonging to PSI students. 

The performance of students on the FE are summarized in 
Appendix B, Sequence 5a,b. Ananalysis of the results shows that the 
PSI students scored significantly higher than the CLR students (p=.005). i 
Thus, it can be infered that the group means are statistically different 


and, since all other relevant variables impinging upon group performance 


** . . ° ° 
Recall that the significance level is the probability of obtaining 
differences in the group means as large as those observed, or larger, 
due to chance alone, when the group means are assumed to be equal. 
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were held constant (except the form of instruction), that PSI produced 
the significant difference observed. The test used to obtain this signif- 
icance level was the non-parametric Kolomogoroy-Smirnov test des- 
cribed earlier [Kim, Jennick, 1970]. As a matter of interest the data 
was also analyzed according to the univariate Fi, 23) test. This test 
revealed the difference to be significant also (p = .004). Thus, if the 
more rigorous assumptions of normality and homoscedasticity are made, 
then the more powerful parametric test rejects the hypothesis that the 
mean scores are equal with a high level of confidence (i. e., 99.6%). 

3. Course Grades (CG) 

The CG for both groups of students is not independent of the 
final exam (FE) score. In the case of the CLR group two "mid-term!" 
exams, of fifty minute duration contributed approximately 50% to the 
grade with the FE contributing the eae 50%. However, the per- 
formance of the students in their classroom meetings weighted on the 
professor's subjective evaluation of each individual. The exact nature 
of the professor's subjective input to the CG of the CLR group cannot be 
measured. However, in no case did the professor's intuitive input to 
the course grade result in its being lowered below that analytic computa- 


tion described below: 


CGcyp ; = -25 (MT) +. 25 (MT) + .50 (FE) 
where 

CGcoyR = course srade of CLR student (i);i= 1,2,... 4, le 

ae = mid-term exam grade on exam (3); jzil,2 

FE = final exam score. 
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In the case of the PSI Group the CG was totally objective. Ifa 
PSI student completed all thirteen required units he merited, automatically, 
a CG of B- (2.66). If he chose to take the final exam (FE) he could im- 


prove his CG according to the scale described below: 





Below 75% means a B- 70.75 "80. 85) 9900 100 % correct on FE. 

The performance of the students as measured by CG is presented 
in Appendix B, Sequence 5 a,b. An analysis of these reste according to 
the K-S test shows that the mean grade of PSI students is numerically, but 
not significantly, higher than that of the CLR group (p = . 146); but accord- 
ing to the univariate F-ratio test this difference is statistically significant 


at p= .025. * The fact that the CLR group mean 


“It is a rather common occurrence for non-parametric tests to judge 
differences in observed means to be less significant than parametric tests 
judge the same differences. This is so because parametric tests have 
greater power derived from their reliance upon assumptions of a partic- 
ular distribution to the data and homoscedasticity. If one is willing to 
make the assumptions necessary to employ the parametric test then Ho 
San be rejected at the .975 confidence level. 
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score on CG is closer to the PSI group than was their mean FE score is 
explained by the boost that some CLR students received as a result of 
their extra credit work and the subjective evaluation of the course 
professor. It is not felt that this subjective evaluation, where applied, 
was unjust. However, it is apparent that the PSI group, as a whole, 
would have appeared even more impressive vice the CLR group had no 
subjective input been added to the formulation of the CLR students! CG. 
4. Rate of Progress (RP) 

The rate of progress is a variable that represents the average 
number of units that a PSI student completed per "work day" (six work 
days per week) throughout the course. Its tabulation here is intended to 
illustrate the effect of the self-pacing aspect of PSI. For comparison 
purposes it is assumed that all CLR students progressed at the same 
theoretical rate during the quarter. Obviously this assumption ignores 
the fact that not all CLR students "learned'' the material equally well, 
that some were pushed too fast while others were, perhaps, "held back. "' 
It further ignores the fact that the professor did not spend equal amounts 
of class time on each topic within his course. However, it was not 
practical or even possible to get a measure of the actual RP of the CLR 
group and the assumption of linear RP must suffice for comparative 
purposes. 

The results show that all of the PSI students completed the 
material not later than the eleventh week of the quarter. Five of them 


completed the course (excluding the FE) in the tenth week. A graph 
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depicting the relative rates of the two groups is included in Appendix B, 
Sequence 6. It is of special interest that the CLR group (when interviewed) 
indicated that their actual rate of progress was not linear at all. In fact 
they spent too much time on mechanics early in the course and were 
forced to rush through more difficult theoretical material presented 
towards the end of the quarter. This revelation, while perhaps disturb- 
ing, is evidently quite common in CLR classes. The phenomenon is 
attributable to perhaps several variables in a CLR course, one of which 
is its absolute reliance on the professor as a primary source of informa- 
tion, as well as an interpreter of the textual materials. Therefore, in 
a CLR course, the professor sets an "average pace" that seems to him 
realistic for "most" of the students enrolled in the course: a pace that 
allows for enough time to cover all of the critical material as required 
by the course syllabus and yet avoids spending too much time on material 
that is not particularly difficult. However, this is never a trivial task, 
and if the professor is concerned about student abilities to satisfactorily 
understand the more difficult course materials he may feel compelled to 
spend more time on the earlier fundamentals. On the other hand, ina 
PSI course, the stress is on the written word so the student can, if he 
wishes, plan his progress from the first day. If the student needs to 
spend considerable time on the fundamental ideas at the beginning 
(because of deficiencies in his background work) he has the opportunity 


to do so (with the aid of the textbook, study guide, and so forth which 


55 





are all carefully prepared in advance, as well as any individual aid he 
may require of the professor), and this is not at the expense of other 
students taking the course for whom many of the course fundamentals 
are of a review nature (thereby leaving more time for these students to 
study the more difficult materials and, perhaps, finish the course ina 
lesser amount of time). 

The variable RP of each student in the PSI class can present 
logistical and administrative problems. For example, all course 
materials must be available when a student is ready for them. It is, 
therefore, incumbent upon the course professor to ensure that the course 
is intact at the beginning or that he is well ahead of his most advanced 
students. Further, it is possible that very good, highly motivated stu- 
dents with few outside distractions may complete a PSI course, evena 
difficult one, very quickly. This has great advantages if the student is 
enrolled ina flexible curriculum. However, in most institutions, operat- 
ing on the concept of an academic year, new courses are not available to 
students at arbitrary times during the term. It is well for the administra- 
tion to consider making allowances to accommodate exceptionally gifted 
Or unusually slow students. In order to assist in planning and predicting 
student completion times the reader is referred to the work of Wagner, 
et al [1973]. In that paper the authors develop predictive models that 
provide indicators of the probable length of time a given student will 


require to complete an individualized course of instruction. These 
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indicants can be used to help program students for future courses 
independent of calendar academic years. 
5. Facilitation 

This output measure attempts to reflect the effects, ifany, of 
PSI on the performance of students in their other CLR classes. It is of 
importance to know whether PSI, with its high demand on student body 
time, forces its pupils to pay a price (diminished performance in their 
other courses) for their accomplishments inthe PSI course. It was felt 
that this effect might be magnified within this experiment because the 
PSI group was taking more credit hours, on the average, than was the 
CLR group (i.e., 19.39 hrs vs 17.54 hrs). In addition several of the 
PSI students were taking another course via PSI, FORTRAN PROGRAM- 
MING. The typical PSI student's course load vice that of a typical CLR 
student is presented in Appendix B, Sequence 7a. Asa result of the 
heavy demands upon their time one might expect to see slightly lower 
cumulative grade point averages (GPA) for the quarter, within the PSI 
group thaninthe CLR group. Therefore, it is surprising and significant 
that the PSI students outperformed their CLR counterparts, not only in 
MA 2045, but in their other courses as well. 

The grade point average (GPA), with the effect of MA 2045 
removed, of the PSI group exceeds that of the CLR group by a margin 
that is significant at (p = .074) according to the K-S test and at (p = .075) 


according to the univariate F ratio, see Appendix B, Sequence 7b. 


(1, 23) 
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This finding is extraordinary because the PSI and CLR groups were 
statistically identical over the relevant input variables of Mental Maturity, 
Personality (Motivation), Prior Exposure, etc., which were measured 
before any instruction at NPS began. These findings are in conformity 
with those of Schimpfhauser, et al [1974] at Ohio State University College 
of Medicine. In addition it suggests that PSI, with its requirement for 
mastery and its logical step-by-step progression, may not only imbue 
in the student a sense of achievement and a desire for high standards 
but, furthermore, it may teach him techniques of study that are more 
efficient and more effective than those learned via CLR. Moreover, the 
PSI student may develop a higher degree of testmanship than he held 
previously. After all he must take at least one test per unit, he must 
learn to work quickly and with great accuracy, and he must certainly 
dissolve any damaging fears of tests that he may have harbored 
previously. 
oe Attitudes 

Finally, the output measure that attempts to reveal something 
of the opinions of the students themselves is considered. Several pre- 
vious studies conducted in this regard report that PSI contributes to 
"positive" attitude growth [Kulik, 1975; Stice, 1971; Gallop, 1970; 
Green, 1971]. What exactly is meant by positive attitudes varies among 
studies but one of the clearest representations of this concept is explained 


by Silberman and Parker [1974]. In their report they concentrate entirely 
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upon the measure of attitudes among students after exposure to a course 
in organic chemistry at the undergraduate level. The basic tool they 
employed consisted of a questionnaire that was carefully designed and 
checked for internal consistency, reliability, and validity. Their results 
indicate that overall the attitude of their PSI students toward their course 
and their mode of instruction was significantly more "positive'' (p = .05) 
than the CLR group. This result was observed in spite of the fact that 
both groups were statistically equivalent across seven student character- 
istic variables at the beginning of the course and that the same professor 
taught/managed both classes. 

In order to measure student attitudes in this experiment a ques- 
tionnaire similar in scope and construction to that of Leiberman's and 
Parker's was designed to be administered to both PSI and CLR students 
in MA 2045. The questionnaire (see Appendix B, Sequence 9b) was care- 
fully explained to both groups and was given to them ten days before the 
final exam. They were asked to respond to the questionnaire at their 
leisure, to consider each question carefully, and to return the question- 
naire not later than the day of their final exam. Twenty-two of the 
twenty-five students (88%) responded. Response was voluntary and no 
names were attached to any of the questionnaires, only the group symbol 
was affixed to ensure correct classification of them for purposes of 
analysis. The results of this analysis are presented in Appendix B, 


Sequence 9a. Note that a score close to zero (0) reflects a more positive 
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attitude, the maximum negative attitude score being five (5). It is ap- 
parent that across all forty-three responses to the questionnaire the 

two groups are nearly identical. The PSI group exhibits a more positive 
attitude on 74% of the questions but the difference reaches statistical 
Significance (p = .10) only 21% of the time (9 of 43). However, in every 
significant case except one (question number 23) this difference is in 
favor of the PSI group. 

When all the scores of the questionnaires were subjected to 
discriminant analysis it was learned that the matrix of correlation 
coefficients was too ill-conditioned to allow for the construction of an 
inverse (the determinant of the matrix was near zero). This was be- 
cause many of the students possessed attitude vectors which were 
nearly parallel. To overcome this difficulty and get to the root of the 
attitude question, the mean scores of the groups across the nine ques- 
tions (where the attitudes were determined to be significantly different: 
those questions marked ** in Sequence 9, a,b of Appendix B) were 
analyzed by program DISCRIMINANT. The outcome resulted in totally 
accurate classification of students into their respective groups. That is, 
based upon a student's response on the nine questions alone, the com- 
puter was able to predict, with 100% accuracy, the actual group to 
which the student belonged. 

The conclusion is that the PSI mode of instruction generated 


within its students certain attitudes that tended to be more positive than 


those exhibited within the CLR group. 
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A general summary of the attitude questionnaire results indicate 
that students of MA 2045 under the PSI mode were required to assume 
more active responsibility for their own learning, learned more from 
the course, could better cope with the amount of information presented, 
enjoyed sharing their knowledge with their classmates to a greater 
degree, looked forward to applying their skills to the solution of practical 
problems to a greater extent, and perceived more clearly the objectives 
and goals of the course. 

Of additional interest might be the feelings and attitudes of the 
professor and tutors of the PSI course. These men felt that the course 
had been a total success, not only across the objective MOE's but from 
the point of view of the personal satisfaction they derived from their 
association with their students. The profes _ indicated he enjoyed his 
role in both courses. He lectured more via CLR but his role was more 
tutorial and highly individualized under PSI. 

7. Summary 

If one considers together only the output measures which are 
devoid of subjective components, thatis, FE, CG, and GPA (with the 
effect of MA 2045 removed) the picture presented is a theoretically 
balanced meshing of student performance measures that have been part 
of traditional education for the last half-century. These three measures 
were analyzed singly in earlier sections. It was also considered ap- 


propriate to examine their combined effects. First, through Analysis 
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of Variance (ANOV) and then via discriminant analysis. The results of 
the ANOV are included in Appendix B, Sequence 8. They reveala signif- 
icant difference across the variables at the p= .01 level, "between the 


groups." 


This indicates that the hypothesis that the two groups per- 
formed equally well can be rejected with a high level of confidence: The 
PSI group did display greater ability and experienced a marked facilita- 
tion effect as a result of their exposure to PSI. The observed significance 
of the cross effect (that is, output measures across groups) resulted from 
the varying degrees of performance difference demonstrated within each 
group across the variables. 

When the output measures (FE, CG, and GPA) are subjected to 
discriminant analysis the computer is capable of properly classifying 
students into their respective groups 80% of the time. If the groups were 
essentially equal, we would expect the computer to classify properly 
only 64% of the time (this is because the prior probability of a student 


being from CLR is .64 = od 


25, based upon the size of the CLR group). 
It is clear that the PSI group demonstrated superior perform- 
ance across all output measures (MOE's). These findings are in com- 
plete agreement with those of the previous studies mentioned earlier 
and lead to the conclusion that PSI, as conducted at NPS during this 
experiment, is an effective method for teaching MA 2045 and is probably 


more beneficial for students than is the CLR technique currently 


employed. 
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IV. DISCUSSION - EXTRAPOLATION ~- FURTHER STUDY 


It is apparent that the experiment, as designed and conducted, was 
successful: it produced meaningful results and accomplished its purpose. 
The experiment leads one to conclude that PSI is an effective and efficient 
method for teaching MA 2045 at NPS. It is efficient in that its students 
progress, onthe average, more rapidly and with better results than the 
standard CLR students. It is effective as evidenced by the facilitation 
effect produced and the demonstrated performance on the final examina- 
tion. To discover these things is important in itself. However, at 
least three questions remain unanswered with respect to PSI at NPS: 

1. Will students who take PSI courses be able to 
transfer their knowledge to other related 
courses with relative ease? 
2. Will students who learn via PSI retain the in- 
formation they assimilate for a longer time 
period than those who learn via CLR? 
3. What other courses, besides MA 2045, are 
adaptable to PSI and on what scale can PSI 
be taught at NPS? 
Questions 1 and 2 could be answered by conducting another experiment: 
One could simply study the transfer and retention effects of PSI on the 
same students used in this experiment. Several studies [e.g., Corey 


and McMichael, 1971; Moore, J. W., etal, 1973] already attest to the 


noteworthy benefits students derive in these important areas and there 
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is little reason to suspect the same results would not accrue to NPS 
officers. 

The third question, however, deserves some special attention. 
Imbedded within it is an implicit issue: Should or could an entire curric- 
ulum (or some percentage thereof) be taught via the PSI mode? Only 
time and experience can accurately provide the answers. There are, 
however, some important indicators of probable results and potential 
benefits to be derived from an attempt to learn the truth of the matter 
concerning PSI. The interested decision maker would be well advised to 
read the admonitions contained in Green's "Fifteen Reasons Not to Use 
the Keller plan, '' Sherman's "PSI Some Notable Failures,'' and Green- 
spoon's "Should An Entire College Curriculum be Taught by the Keller 
Method" [All in Sherman, 1974]. Contained in these papers are the hard 
facts gained from many trials by dedicated men of various disciplines in 
their efforts to hone PSI to its present edge. Even they admit that the 
full potential of the Keller system is yet to be realized. Of special note 
is the fate of PSI at the hands of Massachusetts Institute of Technology 
(MIT) [Friedman, et al, 1975]. This paper describes the decision to 
discontinue PSI in Physics. In spite of small scale successes in intro- 
ductory physics courses, the transition to large scale teaching was mis- 
managed. The endeavor was discontinued after the fourth trial and 
large-scale physics instruction reverted to the traditional CLR mode. 


It is not the purpose here to summarize the Friedman report. What is 
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necessary is to point out the importance of detailed management planning 
prior to the adoption of large scale PSI. In fact, the importance of such 
planning cannot be overemphasized. 

One must examine the nature of the course(s) to be taught. The 
literature is replete with successes in many disciplines (e.g. Math- 
ematics, Chemistry, Physics, Psychology, Electricity and Magnetism, 
etc.). However, the courses have largely been "introductory" in nature. 
This means that they have dealt with the basic principles, mechanics, 
language, and foundations of the discipline. There is no reason to believe 
that ''advanced'' courses cannot be taught via PSI, but since no PSI course 
can succeed without the presence of qualified and dedicated tutors, if in 
an advanced course the tutors are incapable of doing their job because 
their expertise is not up to the task, then the students cannot turn to 
them for the meaningful counsel they will require. 

Given that the subject matter is teachable via PSI, and that a profes- 
sor and tutors are available, then the next step is to write the PSI course 
itself. This procedure is not trivial, requires the existence ofa gifted 
and very knowledgeable author, and can be time consuming. This man 
must be thoroughly familiar with PSI, its principles, objectives, and 
characteristics. Once the course is written it must be validated, at 
least ona small scale. (The course that is the subject of this paper was 
presented twice prior to the experiment and was revised to improve it 


after each trial. The course matured with age and was taught by three 
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different professors and six tutors during this maturation/validation 
process. It is without question now ready for presentation to large 
groups.) Additional work must still be accomplished prior to implemen- 
tation. Materials must be printed, tutors must be obtained and trained, 
and a professor who is totally familiar with his role as manager ofa 

PSI course, must be selected. If this sort of preparation is made with 
additional management planning, then large-scale teaching of PSI can 
follow. It will provide its students with all of the benefits experienced 
by those of smaller groups and the same sort of efficiency and rate of 
progress curves would be expected. 

If, in fact, an entire curriculum is to be taught via PSI thena 
complete rethinking of the nature of the "academic year" is in order. 
Since students will finish their courses at varying times, they should be 
able to continue with the next course in sequence without having to wait 
for the end of an arbitrary ''quarter'' to end. Careful consideration 
would also need to be given to the number of courses a student should 
take simultaneously: PSI courses require intense and diligent study. 

If too much is expected of a student he may tend to do less than if he 
perceived his work load to be manageable and realistic. This thesis 
does not address these issues. However, it is obvious that such matters 
must be carefully considered prior to any decision to expand or extra- 


polate PSI at NPS beyond the "small-scale" level. 
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V. SUMMARY AND CONCLUSIONS 


A carefully controlled experiment was designed and conducted for 
the purpose of determining the relative instructional effectiveness of 
PSI at NPS. It was felt that the results of the experiment would provide 
valuable information pertinent to the educational processes managed 
under DOD authority in general. The vehicle for the study was MA 2045 
(Computational Linear Algebra). The course was written in its PSI form 
by Professor Maurice D. Weir. Prior to the conduct of this experiment 
the course was tested twice and carefully revised following eaee trial. 

Once it was subjectively established that the PSI course was valid, 
two groups of students (m = 12; n = 16) were selected at random and 
taught MA 2045 via PSland CLR. The same professor conducted both 
classes. Both groups were screened, via testing, prior to any instruc- 
tion at NPS to lend credence to the hypothesis that both groups constituted 
independent, random samples drawn from the same population. 

The two groups were then given the linear algebra course. The 
output variables, (MOE's) were: 
percentage of students completing the course 
final examination scores 
course grades 
facilitation 


rate of progress 
student attitudes. 
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These output variables were subjected to several statistical tests 
(parametric and non-parametric) and a sophisticated data analysis tech- 
nique (Discriminant Analysis) in order to determine the significance of 
the distance of the group centroids in reduced space. 

Group performances across variable one (above) were equivalent. 
Across the other variables, however, PSI students demonstrated a 
statistically. significant superiority compared with CLR students in 
MA 2045 at NPS under the conditions described inthis paper. Asa 
result PSI is considered more effective and more efficient than CLR for 
teaching the course Computational Linear Algebra, MA 2045. It is con- 
sidered valid to extrapolate beyond this specific course and to apply the 
results of this study to similar courses of similar scope in other areas 
of mathematics and other disciplines as well (see Appendix A). If, 
however, the results of this study are to be extended to a larger scale, 
then it is strongly recommended that the admonitions and considerations 
explained in the discussions section of this report be taken to heart. If 
an entire curriculum is to be constructed around PSI then careful analysis 
of the appropriateness of the structure of the traditional academic year is 
warranted. 

It is recommended that further study to determine PSI's effect on 
retention and transfer be conducted. Finally, the relative cost-effective- 


ness of PSI for NPS should be documented. 
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ARPENDDA 


Student Performance ina PSI Course in Networks Flows and Graphs 
(OA 4633) 

It is the position of adherents of PSI that the teaching mode is valid 
for many types of courses, at all levels across a broad spectrum of 
academic disciplines. The main body of this report discusses an experi- 
ment designed to measure PSI's effectiveness vice CLR and employed 
MA 2045 as a vehicle. Concurrently a course in Networks Flows and 
Graphs (OA 4633) was being taught by Commander Joseph Cyr, Curricular 
Officer, Operations Research, Systems Analysis. Commander Cyr, like 
Professor Shorb, had attended a workshop on PSI and had been interested 
in converting his Networks course for presentation via PSI. By Quarter 
IV FY 75 his materials were ready and he permitted his students (12) to 
choose the mode of instruction (PSI or CLR) they preferred. They were 
all MS (OR/SA) candidates in their seventh quarter of an eight quarter 
sequence. They had signed up for OA 4633 as an elective. 

Eight of the students opted for PSland four for CLR. Prior to 
instruction the two groups were compared across their Grade Point 
Averages (GPA), Mental Maturity scores and Personal Inventory scores. 
Although with small sample sizes, statistical ee little mean- 
ing, the group means were very close across all variables (K-S, non- 
parametric test of equality of group means and suggested that there was 


no reason to reject the hypothesis of co-location of group centroids. 
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Commander Cyr selected tutors (2) for his PSI group and conducted 
his course with these students according to the principles of the Keller 
Pian. 

At the completion ofthe course the students were compared across 
the output measures generated by the final exam, the rate of progress, 
and the student attitude questionnaire. The results are in complete 
agreement with those found in the main experiment discussed in the body 
of this report. Again, it is not meaningful to infer statistical significance 
with such small samples. However, the success of the course and the 
impressive performance of the PSI students confirms the viability of 
teaching OA 4633 via the Keller Plan. 

This fact supports the position taken in the body of this paper that 


PSI will work effectively at NPS on courses other than MA 2045. 
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APPENDIX B 


Synopsis of Data Analysis 


This section of the thesis is intended to epitomize the data collec- 
tion, analysis and results of the experiment conducted to determine the 


relative effectiveness of PSI vice CLR for teaching MA 2045 at NPS. 


foe PRIOR TO THE INSTRUCTION 

1. After two groups of students were scheduled for instruction via 
one mode or the other (selection determined by NPS administrative 
offices against internal scheduling criteria) it was necessary to conduct 
certain tests to support the assumption of the existence of matched 
groups. A total of twenty-eight subjects were available for the purposes 
of this experiment. Twelve students constituted the PSI group (m = 12). 
The remaining sixteen individuals comprised the CLR group (n = 16). 
Measures of six student characteristics were obtained according to the 
methodology discussed in the main body of this thesis. The raw data is 
presented in Sequence l. 

2a. The next step was to investigate, using statistical inference, 
the hypothesis that the two groups constituted random independent 
samples drawn from the same population. Consider the possibility of 
characterizing each student according to the six dimensional vector 
constructed from his performance on the characteristic tests described 
in the main body. In general an individual student is described by the 
vector: 
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6 

The set ft Tel, s.au0, 12} constitutes a data base as does the 
setix,ii=13, peneke 28 | 

The space is of dimension six and the hypothesis to be tested states 
that any observed differences in the location of the centroids of the two 
groups is so small as to have reasonably occurred by chance alone. The 
first ''test'' of the hypothesis (Ti) consisted of the subjective inspection 
of the historgrams generated by plotting the frequency distribution of 
each component of the vectors (x, ) singly. Since these histograms all 
assumed the same shape, over their respective ranges, it seemed 
reasonable to continue with another test. 

The test selected was the K-S non-parametric two-sample test of 
the homogenity of cumulative distribution functions (one-way test). This 
test was applied across each component of the vector sets one at a time. 


The results of this analysis are portrayed in Sequence Za. 
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2b. Since the power of any test is realized when rejection of Ay is 
possible, at an acceptable level of confidence, it was decided to test the 
null hypothesis with a more powerful parametric test. This test, the 
univariate F() 26) ratio required the assumptions of a normal distribu- 
tion underlying the random variables (si and equality of variance within 
the samples across the variables. If either of these assumptions are 
invalid then the reliability of the test is questionable. However, it is 
known that the F-ratio is robust with respect to these conditions and 
since human performance is often normally distributed the test was 
employed. Sequence 3 displays the outcome of this analysis. 

3. Both the non-parametric analysis of Sequence Za and the para- 
metric investigation of Sequence 3 considered the six components of the 
student characteristic vector one variable at atime. In order to look 
at all variables taken together a 2-way, nested factorial ANOV was per- 
formed on the data. The results are presented in Sequence 4. 

The assumptions which are made when employing ANOV are the 
same as those of the univariate F-test (i.e., independence, normality, 
homoscedasticity). Fortunately, ANOV is also robust with respect to 
these assumptions and its results are in complete conformity with that 
which was expected, based upon the results of the previous testing. We 
find that there is no significant effect between students within groups and 
there is no difference between the groups themselves. The significant 
difference between variables (characteristics) is due to the various 


scales of the scores and 
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the cross effect is due to the fact that one group did not perform consist- 
ently better or worse than the other across all measures of effectiveness. 
4. The final investigation of the data involved the use of a technique 
called linear Discriminant Analysis (DA). This technique is employable 
when one assumes that the discriminating variables have a multivariate 
normal distribution and that they have equal variance-covariance matrices 
within each group. In practice, the technique is very robust and these 
assumptions need not be strongly adhered to. Discriminant analysis 
begins with the desire to statistically distinguish between two or more 
groups of cases. In order to accomplish this task discriminating var- 
iables that measure characteristics of the groups are selected. The 
mathematics of the method are such that the variables are weighted and 
combined (linearly) so that the groups are forced to be as distinct as 


possible. The "discriminant" functions are of the form: 


= = qi, 7 teow nme a5 
where: 
D. : the score on discriminant function i 
i 
dd. : weighted coefficients 
J 
Z. +: standardized values of the variables. 


In this analysis one such function could be constructed since only two 


groups were examined. 


Once the function has been constructed analysis and classifica- 


tion can proceed. 
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In the analysis several tools are available to interpret the data; 
that is, to test for the success with which the variables have discriminated. 
The function itself is an axis in the variable space (six-dimensional) and 
it is used to study the spatial relationship between the groups. The co- 
efficients are analogous to those derived in multiple regression or factor 
analysis. They serve to identify those variables which contribute the 
most to differentiation along the discriminant axes. 

In the classification phase one attempts to place students into 
their respective group based upon their function score. If the vanes 
are valid discriminators then we would expect proper classification a 
"high"' percentage of the time. 

The results of the discriminant analysis of the two experimental 
groups across the six variables of classification reveals successful dif- 
ferentiation is achieved; in that, proper classification of students into 
groups takes place with 78.57% accuracy. The overall significance of 
the separation of the group centroids allows for rejection of Ao with 


95.3% confidence. This level is achieved principally because of the 


variable: ''prior exposure.'' This score was derived from the algorithm 
below: 
m 
PE, z a» [ (cg won} S 
where 
PE; : the prior exposure score for the jth student 
c : a number assigned to a particular course which the jth 


student took prior to matriculation at NPS 
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W =: a weighting factor 


N; : the number of years since the je student took course i 
Ss : a scaling factor to ensure 7s: is greater than or equal 
to zero 


-th 

For example, suppose the j” student had taken college algebra ten 
years ago, linear/matrix five years ago and linear programming four 
years ago. His PE score is derived as follows: 

college algebra is worth one point 

linear/matrix algebra is worth three points 

linear programming is worth six points 

W is equal to one-half; S equals five 


thus 


ae 3 


[1-1/2(10)}] + [3-1/2(5)) + [6-1/2(4)] + 5 


5.5 
It is felt that this algorithm gives an indication of the relative PE of the 
students but it is arbitrary and its validity is questionable. To measure 


the effect of this variable (taken singly its F ratio is 2.535, signif- 


(1, 26) 
icant at p=. 120) on the discriminant analysis as a whole, it was removed 
from the analysis. When this is done the significance of the distance 
between the group centroids drops markedly. The probability of error 
if H, is rejected goes from (p = . 047) to (p= .671). Thus, if PE is 
removed from the analysis the two groups are practically indistinguish- 
able. The percentage of students properly classified drops to 64%, 


which is close to that which should be expected due to the computer's 


knowledge (a priori) of the sizes of the two groups (i.e., LOVER =o 57"). 
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In any event the average PE score for the CLR group exceeds that of 

the PSI group. As such, if any advantage is presented it exists within 

the control group and not with the experimental group. This fact, coup- 
led with the knowledge that the PSI group labored under a larger credit 
load than the CLR group forces one to view the results of the experiment 
as indicating marked advantages for PSI as a teaching technique over CLR 


as measured along the output variables. 


meee Os lT-INSTRUCTION ANALYSIS 

1. Final Exam Score: the performance of the nine students in the 
PSI course and the sixteen students in the CLR course is presented in 
Sequence 5a. Sequence 5b reveals the significance of the difference in 
group means according to the K-S test and the univariate F-test. 

2. Course Grade: the performance of the students by group as 
measured by course grade is presented in Sequence 6a. Sequence 6b 
reveals the significance of the difference in group means according to 
the K-S test and the univariate F-test. 

3. Rate of Progress: the average rates at which the two groups 
progressed through the course material is depicted in Sequence 6. The 
dots which have been plotted display the graph of the average number of 
units passed by the PSI group against days available in the quarter. The 
dashed line is the theoretical rate at which the CLR students, as a group, 
progressed through their course. The solid curved line shows the rate 


at which the CLR students perceived their group rate of progress. Note 
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how shallow the slope of this curve begins. Towards the end of the 
quarter it becomes steep, indicating the presence of a rush to complete 
syllabus material prior to the final exam. 

4. Grade Point Average (with contribution of MA 2045 removed): 
The typical course loads, with associated credit hours, taken by the two 
groups is shown in Sequence 7a. The grade point average (GPA) achieved 
by students within the two groups were analyzed according to the K-S 
test and the univariate F-ratio. The results are summarized in 
Sequence 7b. 

5. ANOV: The model used to examine the output measures FE, 
CG and GPA taken together is presented in Sequence 8. 

6. Attitudes: Sequence 9a displays the different average attitude 
scores computed for each group on each question of the attitude survey. 
The range of scores is [0,5] with a lower score indicating a more 
positive attitude. Sequence 9b presents the exact average score of each 
group on each question. 

If one looks at the results of this survey across all questions 
taken together the conclusion that the two groups are indistinguishable 
is supportable by discriminant analysis. However, when one restricts 
attention to those nine questions (**) where the difference is so large as 
to be significant at the 90% confidence level it is found that the two 
groups are completely separable via discriminant analysis. That is, 


the students can be properly classified into their proper group with 
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100% accuracy. Of these nine questions only one (question #23) reveals 


a more positive attitude accruing to the CLR group. The conclusion 


is that the attitude of PSI students is at least as positive as CLR students 


but that across eight specific points it is demonstrably superior. 
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Data Obtained from Tests to Measure Characteristics 
of the Student Groups 


VAROOI] 
Sy) 0.0 
So E20 
S3 z.0 
Sy 2.0 
Ss 2.0 
ge z2.0 

O Sz 3) 
Sg D0 
510 370 
Si FZ 0 
S12 13 
Gj 4.50 

é. 4.27 

2 Ve 

s i) 

“ Sia 1.0 
Sis r.0 
S16 1.0 
Si7 ate 
Sig i, 0 


VARO02 


47. 


48. 


50: 


43. 


49. 


Do. 


Sor 


48. 


53. 


Doe 


5S: 


51. 


0 


0 


0 


SO sco 


oe 


47. 


50. 


44, 


54. 


43. 


Drks 


Al 


0 


Sequence l 


VARO03 
47. 
50. 
aN 
55. 
48. 
49. 
43. 
49. 
43. 
54. 
49. 


SI. 


0 


47.91 


4.98 


45. 


48. 


40. 


40. 


44, 


43. 


0 


0 


0 


0 


0 


0 


80 


VAROO04 


85. 


80. 


Bin 


aT. 


63. 


59. 


68. 


Sze 


66. 


85. 


ile 


{fei 


0 


0 


0 


13.200 


47. 


Ut 


1 


A 


Te. 


18. 


Ie 


0 


Ol 


VAROO5 
83.0 
v2.0 
47.0 
16.10 
12.0 
Syei0 
35.0 
65.0 
60.0 
81.0 
19.90 
170 


68. 75 
El255 


67.0 


120 
64.0 
69.0 


Taa0 


VARO06 


Ss 


0 


29 


1.30 





Sequence 1 (continued) 


VAROO1 VARO0Z VARO03 VARO04 VAROOS5 VARO06 


Si 20 male 54.0 58.0 53.0 4.2 
So ZG 50.0 36.0 69.0 56.0 5.4 
So 3.0 51.0 48.0 66.0 66.0 Aa 
S5> 3.0 49.0 50.0 82.0 78.0 6.4 
a.S23 3.0 57.0 46.0 68.0 7220 5.6 
6 S24 550 53.0 55.0 89.0 77.0 6.0 
So. 6.0 50.0 37.0 e250 73.0 8.2 
So 720 55.0 46.0 e260 110 7.8 
S54 TAO 55.0 53.0 82.0 7210 6.6 
S29 9.0 59.0 54.0 78.0 82.0 7.6 
x, 3.25 50.56 46.06 74. 68 1Oeae 5. 32 
S; eee si Ol 5.87 10.68 ol 1. 68 
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VAROOL: 


VAROO2: 


VAROO3: 


VAROO04: 


VAROOS: 


VAROO6: 


Sequence 1 (continued) 


The score obtained by a student on the linear/matrix 
algebra pre-instructional inventory exam. 

The range of scores = [0, 20}. 

The mental maturity language proficiency score 
Range = [0, 60] 

The mental maturity non-language proficiency score. 
Range = [0, 60] 

Personality measure (personal traits) 
Range = [0, 80] 

Personality measure (social traits) 
Range = [0, 80] 

Prior exposure score 


Range = [0, 10] 
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Sequence Za 


Variable Test Statistic P(Denn = fs) = Clee) 


Pre-Instruction 
Inventory Exam . 208 . 198 


Mental Maturity 
Language - 292 m098 


Mental Maturity 
Non-Language 213 030 


Personality Traits 
Personal 229 one 


Personality Traits 
Social 229 .278 


Prior Exposure 
Mathematics Courses . 417 . 887 

This table illustrates the great similarity in the performance of the 
two groups across the input characteristic variables. If one looks at the 
mental maturity language score, it is seen that the test statistic is .292. 
The probability that the K-S criterion variable could be greater than or 
equal to this value is .558. This means that if we were to reject the 
hypothesis (H,) of equality of the two groups across ae variable (VARO0Z) 
in favor of an alternative (H,: PSI#CLR) the probability that we would err 
in doing so is (1-.558)=.442. Thus, we cannot reject Ho with any confi- 
dence greater than 55.8%. This is far too low for most statistical/ 
experimental work. Sequence 2b is a computer program which returns 


the test statistic reported and the exact probability of observing a value 
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as large or larger than that which was observed. In order to utilize 
this program the user provides the data and appropriate commands to 
read the input and write the output in the desired format. The program 
was written from instructions in Kim and Jennrich [1970] and adapted 


for use in this thesis by the author and Gerald Learmonth of NPs. 


84 





Sequence 2b 


ey 
o ov Se Bee IO 
As OG 4, «6 ea Y ¢g OV 
~ 2 ae Ss UO Oe 
no = oO oO rae eT Y OC o6. 
ort we + } GS Oe to 4 oH 
~ a 3 a9 — a = © Pe 0 y O 
Bec Ope i Gee Seer dee 
eS $2> Gc 3ERs 
feo lee ae ee 
oma SO jee ae ae cin 
v = — iS SU 8 ry 0, & = 
Bee | Osc len Garg cost nO 
eo Sa ae Se ce 
et «6 v oO & gee Se 
MM ty A, eG . 
QD a =f ec On OU 
wlio vb Ye A omy 8@ HB 
°o we ut a 3 oO & Oo Y 
Hoan eM o-n A aa AHu OB 
A, o = a OO 
n VU Ora a. +4 O “oo om 
4qag”72 VO] g — "go H j 
Se bt eg hog Oreere, ae . 
Fie Ope ew eno ae Ort mcs ly 
— 
Oo 
= < uw 
-~ - 
N + 
* = 
Oo — 
© — D> N 
N = 
— Oo — QE 
aed | -~ ~ ~ Q e -~ Oa < = 
oe ca O=—t ~<_ ~ o oO = ee ae = 
— a => zo NX © z= MN + DO Ke a j=) 
| Sr ewe | ~~ | a ae e- Ww ~ = Hl mi) 4 
~—— - o~p— <f ht <I — OV So ® «A — — ete <f 
=> =~ Oomtx ax — —-2 aS © + = = ae ~ ~ 
~ a -2O~ ow N = U. + = t~w 75 ad ~ 
<— ~ ~ mY IW” - O~w Q—O na QO D> —+t am ot 
—— =z a“ ~uo wm Zz OF OZ + = =I 2= On ~ 
au ~ & ~ ~~ T NZ = <I N~w—m - w= + | = SL) — <— 
=x xm = m N Zz<—~ + — tt oO —9A voz 2 New ame + << e 
oO ww Zw ~ Sieyo 2.4 —@ ij 209 2ZOe Sraezoar 2 ww ~ Zi 
Zz b= ke we ~ — om we ~ = sa Sw ees o of Oe ee) ei Za o mmo 
=o) Cia wt ase = e= WwW < en ZOHO SAO Re DO HD wD “OO gp SY ODMH S UW 
Om ZOOWuwnDt woo hoaetih Wth DO az W2 Ome ell Ht oD RH eH UND bom er tL DD 
MAZHONN RRS a ZNOO toma bm CF) coe Om Ze SS BEZ ANTM ADMZ SIH MeAMZLZ DNAwa QAMRdMIAIeZZZ 
eat a= Ste ee ~AVNAOIUINEHRO de ey FCO atm OUt aaa Oe O2f A wt I Ney 
OWS Wad eee Hb OO LANILL e+ eRLL o+b Il Leas OW mat eS eR Nae ZN ROD VW ZANT Re ee 3; 


ZEUOde MZ AMZN WAAR MH WM ZZ Zw ZRO ZZ SZ OL Rw OS ZR ODZ HON WE TIHZZEO 
PM eZIIOW OO MOTI OL IS HOM IOS HW TULLE ZDay HMOWILOO Iw LOWLO XYWZDRHYO! OW ~OOWZ 
LOAZZOOOANVOXNOO* > LY DOROKN ZO ON DOO OR RO CWUILOY DODSOQ ZDNODNOIYWNAYQYON RS MOO W 


a N ~ Oo uy © -_ N Nea 
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Sequence 3 


Variable Wilk's Lamda Fi, 26) 
Inventory . 9668 . 8929 
Mental Maturity 

Language - 9986 0653 
Mental Maturity 

Non Language ~9 720 . 7760 
Personality 

Personal . 9936 . 1669 
Personality 

Social e 9925 . 1957 
Prior Exposure oe 225555 


This table lists two statistics for each variable difference score. 
The first is Wilk's Lamda (), 0¢A¢1. As A —> 1, the probability 
that the observed difference in means is likekly to have occurred by 
chance alone approaches one, inthe limit. The second is the univariate 
F_-ratio with appropriate degrees of freedom. For this statistic to be 
Significant it must be greater than one. Note that only across 'Prior 
Exposure'' does this statistic approach statistical significance. If we 
were to reject Hy across the variable we would be forced to do so at 
some confidence level less than 90%. <A level that low is normally con- 


sidered insufficient for statistical/experimental work. 
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Sequence 4 
Analysis of Variance (ANOV) to determine significance of observed 
differences in student performance across six-input variables by student 


by group. 


Model 
peg et ©, OGG) > age 


miere: x (ijk) : the observed performance of tie ae group across 
the ve variable by the kth student. 
cco : the grand mean 
(eG. + €(ik)): "Between-student'' effects 
( 1 = PSI 
where: G,: Group i= 
(2 CLR 


Lice « eliZeee ieee 
C.,.: Random error ke 


13, 64 GS): aie =32 
Cj a Gi; 5) +f 2 (ijk) : within student effects 
where: C; = characteristic variable j: j=l,..,6 
Ci ik = random error 


All levels are fixed. 


Legend: 
Source: the entity which could be generating observed variations 
df : degrees of freedom associated with the source 
| SS : seem of squares of observed variates 
MS : mean square 
EMS ; expected mean square 


ui go(V1, v2): the ratio that constituted the test statistic. 
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PSI Group 


CLR Group 


Student 


Sequence 5a 


FE 
Raw Score 


190/200 = 
199 = 
173 = 
ie = 
199 = 
188 = 
192 = 
194 = 


198 = 


i 


189/200 
131 = 
158 . 
179 = 
176 = 
158 = 
200 = 
142 . 
170 = 
180 = 


u59 = 


% 


295 
«Ue 
2 One 
. 86 
- 998 
94 
76 
ois 
og 
o95 
. 66 
oad 
. 90 
200 
219 
. 00 
21 
2O5 
. 90 


. 80 


89 


x = 189. = 
PS] O53 = 95% 


Std. dev = 10.58 


*denotes students who 
took the FE early 


XcCLR = 164.37 = 82% 


Std. dev = 22.02 


* 





Sequence 5a (continued) 


Student Raw Score % 
52] 173 = . 87 
114 = ot 
22 
S>3 151 Sera 
S24 171 = . 86 
oe 179 = 90 


90 


Sequence 5a (continued) 


Statistical analysis of FE data 


H;: there is no difference in the performance of the two groups as 
measured by Final Exam Scores ( Fis). 


Hy: otherwise 


Confidence level = 90% 


1. According to the non-parametric K-S test of the equivalence of 
two samples: 
Test Statistic (T.S.) = .653 
PDs tert 2.) = lop = .99a97..90 


Conclusion: reject 2 the groups did not perform equivalently across 


Pais, 


Zs According to the parametric univariate F- ratio() 26) 
Test Statistic (T.S.) = 10.1017 
P(F(y 26) > 7T-8-) = ep 990 >. 90 


Conclusion: reject H.. 


Summary: the performance of the PSI group is significantly superior 
that of CLR. H, could be rejected with more than 99% confidence wit 
either test. 


oe 








PSI Group 


CLR Group 


Student 

Sy 4.00 
So 4.00 
S. 3.66 
S4 3.66 
S. 4.00 
S¢6 4.00 
S7 4.00 
Se 4.00 
So 4.00 
S19 4.00 
Si 600 
S12 3.33 
S13 4.00 
Si4 3.66 
Sis 3.66 
S16 4.00 
S17 3.00 
Sig B35 
Sig 4.00 
520 o535 


Sequence 5b = 


Course Grade 


(C2G.) 


(A) 


(A) 


p é = 3.92 =~ aCe 
Xoo] 329 (A-) 


Std. dev. = 2.150 


(A-) 


(A-) 


(A) 

(A) 

(A) 

(A) 

(A) 

(B) 

(B+) 

(A) as = 3.56 ; (Bt) 
(A-) Std. dev. = .435 
(A-) 

(A) 

(B) 

(B+) 

(A) 


(B+) 
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Sequence 5b (continued) 


Student CmG 

S21 4.00 ; (A) 
S55 Z 6 Ge; (B=) 
So3 sess”; (B+) 
S24 Sypo (A-) 
Sos aro ¢ (A) 
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Sequence 5b (continued) 


Statistical analysis of C.G. data 


H,: there is no difference in the performance of the two groups as 
measured by the Course Grade (C.G.) 


H,: otherwise 

Confidence level = 90% 

1, According to the non-parametric K-S test of the equivalence of 
two samples 


Test Statistic (T.S.) = .438 
P(Din 2 T-S.) = 1-p =. 854 < .40 


Conclusion: accept ee the groups did perform equivalently across C.G. 


2. According to the parametric F-ratio fie2G) 
Test Statistic (T.S:) = 5.8272 
ee) leo) et ee 


Conclusion: reject Ho: 


Summary: non-parametric analysis denies rejection of H_ at 90% level. 
H. could be rejected at the 85.4% level. Parametric analysis denies 
Hy at 97.5% confidence level. 
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Sequence 7a 


Typical Course Loads of Students By Group 


Course Name 
Intro. to Fortran Prog. (CS 2700) 


& Adv. Rev.,, Circ, Sigs, cys, (BH 210G) 


ro 


O Digital Machines (EE 2810) 


PST 


Computational Linear Algebra (MA 2045) 


Differential Equations 
Total 


Course Name 
Elec. Engr. Fundamentals J (EE 2104) 
Calculus and Vector Analysis (MA 1100) 


Computational Linear Algebra (MA 2045) 


CLR Group 


Rev. Vector Mech., Fluids (PH 1051) 


Programmable Calculators (ST 1810) 
Total 


96 


Credits 


Credits 





Sequence 7b 


GPA (w/o MA 2045) 


3.76; (A-) (Std. Dev. 


3.42; (Bt) (Std. Dev. 





Statistical Analysis 


a Group GPA's are equivalent 
H, : otherwise 


confidence level = .90 


1 K-S Test: 


= TS. = .9260 =.40 


Zz Univariate F-ratio() 26) 
Gao. = 3.358 


P( = T.S.) = .925 >.90 


F(a, 26) 


Summary: reject H, 


vif 


574) 
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Sequence 9a (continued) 


CLR Group Ave Response 
PSI Group Ave Response 


Questions on which discriminant analysis of attitudes was 
based 


Significance of difference such that hypothesis of equality 
rejected at .90% confidence level. 


Question #23 where difference in favor of CLR group was 
statistically significant (also included in discriminant 


analysis). 


Trend questions. 
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Sequence 9b 


Attitude Survey Questionnaire 


Mean Score * Significance 
Question PSI CrR Eas. 10{1, 29) 
1. Memorization of basic facts in this 
course, is sufficient to obtain a 
"B" grade. 355 3. OF 0.20 
2. I would recommend that a friend 
take this course under the instruc- 
tional technique I was exposed to Z. 88 2.92 
(i.e., PSI or Conventional). 0.00 
3, This course required that I assume 
active responsibility for my own 0.22 1. 84 14,5755 
learning. 
4, Prior to this course my study 
habits were Ll. 7% 2.53 lot 
5. Right now my study efficiency 
is Sil ato 0.28 
6. I feel that the subject matter of 
this course was difficult to 2.66 3.23 0. 83 


understand. 


7. Iam not satisfied with my 
understanding of the content Zoe 2.69 0.48 
of the course. 


8. This course encouraged the 
development and use of my 2.88 2.46 0. 42 
reasoning skill. 


9. Iwas often frustrated in my 
efforts to understand the 3.88 3.30 Onno 


course material. 


10. Ihave learned very little 
from this course. 1.55 Zoe eet 
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me. 


Va 


eS 


14, 


DD. 


16. 


re 


18. 


19. 


20. 


21. 


ae. 


Taking this course has increased 
my interest in the subject area. 


The instructional technique em- 
ployed in this course is an effec- 
tive method for teaching the 
subject matter. 


I felt a keen sense of competition 
with my classmates. 


My main reason for working hard 
in this course was to obtain a 
grade of ''B" or better. 


Too much material was covered 
in this course. 


I had difficulty getting assistance 
when I needed it. 


If I was asked for clarification of 
a point by a classmate, I felt 
very good about giving him 
assistance. 


This course made me want to 
explore the subject area in 
greater depth. 


The instructional technique 
employed encouraged coopera- 
tion with my classmates. 


I found that taking this course 
was an unpleasant experience. 


I look forward to applying the 
techniques I learned on 
Practical Problems. 


Most courses, in similar 
subjects, should be conducted 
as this one was. 
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Mean Score 


Psi 


1.00 


1.00 


On2zZ 


ae 


Yo 


3.22 


Lecture 


3223 


2.69 


p=.1 


0235 


Significance 
F gis 20) 





23. 


24, 


25. 


20. 


al. 


es. 


29. 


30. 


ol 


52. 


33. 


34. 


The amount of knowledge I acquired 
was not worth the effort I expended. 


In retrospect, taking this course 
was a worthwhile endeavor. 


I saw this course as an opportunity 
to learn rather than as a require- 
ment to be met. 


The information presented in this 
course will be of value to the 
solution of practical problems. 


Knowing what I do now, I would 
prefer to have taken this course 
via a different instructional 
technique. 


The conduct of this course 
accommodated my individuality 
(e.r., personal abilities, ex- 
ternal distractions, conflicting 
commitments, etc.). 


The instructional technique did 
not foster my own desire to 
learn the material. 


The objective of this course 
were made clear. 


Frequently I felt bored with 
this course. 


My ability to grasp the subject 
matter exceeded my expectations. 


The objectives of this course 
were met. 


The course was of sufficient 


scope to challenge my academic 
abilities. 
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Mean Score 


Pel 


3.33 


1.44 


2.66 


1,44 


3.23 


30 


aso 


Significance 
Lecture F (1,20) 
p=. 10 


nos 


294 


2 ke 


ale Je 
op 


w16 


. 85 


. 34 





aD. 


36. 


oe 


38. 


ro. 


40. 


41, 


42, 


43. 


Generally speaking, I felt I was 
pushed through the course ata 
rate faster than would allow me 
to absorb the material. 


The primary motivating factor 
during this course was my own 
desire to acquire knowledge. 


I feel confident in my ability 
to do well on the final exam. 


This course, inits present form, 
could be taught effectively by any 
instructor qualified in the subject 
area. 


I believe that I have mastered 
the course material. 


When I began this course my 
prior attitudes towards tests 
were 


As the course progressed, my 
attitudes towards tests 


My prior motivation to learn 
this material (i.e., before the 
course even began) was 


As the course progressed, my 
motivation seemed to 


Mean Score 
Lecture F (1210) 
p=. 10 


Psi 


1.88 


2.66 


2.46 


2.69 


3.07 


2.92 


2.92 


A positive attitude is reflected by a lower score 


Indicates that the observed test statistic ae 


at the . 10 level of significance. 
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Significance 


quer 


1. 43 


E in? 20) is significant 
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